Adobe Firefly for Video Announced – Bringing Generative AI to Creative Cloud

Software NewsNAB 2023

by Mascha DeikovaApril 24^th, 2023

As we know, NAB is not only a great place for the industry to meet up, but also a perfect time for companies to introduce something special. Adobe is no exception. After announcing new features for Premiere Pro like text-based editing or automated color tone-mapping, they also revealed their future plans for implementing Adobe Firefly for video, hugely expanding their family of creative generative AI models. What to expect and why does it sound exciting yet unnerving? Read all we know about it below.

First things first. Adobe Firefly is an AI-driven set of tools, based on Adobe Sensei – the company’s artificial intelligence and machine learning framework. Firefly was introduced to the world last month and is now in an active beta-testing phase. At the moment, it mainly allows creating images from text prompts (the offer for such generative AI tools is already massive – as an example, we wrote about Midjourney and Stable Diffusion here). In addition to this, Firefly offers the possibility of generating text effects – an interesting toy for designers and illustrators.

At the NAB 2023, the company disclosed that moving forward, they’re going to bring generative AI into Adobe’s video, audio, animation, and motion graphics design apps. Some features in the making that have already been presented are generating music for clips via a simple text description, or changing the color scheme of your video in one click. Intriguing? You bet!

Adobe Firefly for video: what to expect

The range of concepts that Adobe explores in the AI arena stretches from storyboards to the final editing. Let’s start with pre-production. As stated in the official press release, creators will be able to upload their script, let the deep-learning engine analyze the text, and then automatically create storyboard scribbles. In the video presentation, this process literally takes just a single click of the mouse and you will get something like the following images:

Amazing, right? And that’s not all. Another click on the button “Make previz”, and voilà! – you get a simple, yet detailed animated pre-visualization of your film right in the timeline.

I guess if this Firefly feature is so simple and user-friendly, it will be of enormous help to aspiring filmmakers or independent productions. At the same time, this development direction seems worrisome to me. There has already been a lot of discussion about the potential impact of generative AI on the future of human artists. Some creators assume big companies might take advantage of tools like this instead of hiring professionals. One case is how Netflix used AI to create art for an anime short, justifying it with the “industry labor shortage”.

Useful features to speed up different workflows

Setting aside this ongoing debate, Adobe Firefly may become very useful in accelerating several workflows, especially in the social media and corporate video production area (I write “especially”, because that’s where time becomes your best friend, and some processes don’t have to be done in full cinematic quality).

Let’s take as an example the before-mentioned text-to-color enhancement. This feature will allow creators to change color schemes, time of day, or even seasons in their video clips almost instantly. To do so, they will only need to write a simple prompt, as in the example below:

Yet another AI-based tool that has been announced will not only transcribe an interview but also identify the keywords and even suggest relevant B-roll clips, making a rough cut for you.

When we can integrate Firefly into other Adobe applications, we will not only be able to generate text effects (like in the beta phase) but also animate them. Creating stunning fonts, title cards, graphics, and logos is the next step, the developers promise.

Adobe Firefly for audio

Another area for exploration of generative AI is, of course, audio. Here, Adobe also presents a couple of new concepts for Firefly. First, their presentation shows how creators will be able to ask artificial intelligence to compose a custom music track for them. A couple of keywords, and here you are.

Afterward, you may use an advanced sound effects feature and let AI analyze the content of your video, looking for matching sounds. As developers emphasize, all generated melodies and tracks will be royalty-free, so the creators can add them to the final cut.

The biggest appeal of Adobe Firefly

And that drives us to the biggest difference between Adobe Firefly and other generative AIs (at least, in my opinion). The company promises that they train their models only on legitimate and inclusive datasets, like Adobe Stock images, openly licensed content, and public domain data where the copyright has expired. This is quite a new and definitely the most conscientious approach compared to some other developers of neural networks, who use everything they can get their hands on.

What this means is that content generated by Firefly will be safe for commercial use and solve part of the attribution problem. However, I also read some other opinions, where filmmakers, for example, ask Adobe to give Stock creators the freedom to choose whether they want their footage to be used for the Firefly model training or not (and if so, to get extra compensation for it).

How to try these new AI tools out

As for now, Adobe Firefly is available in beta, and anyone can request early access here. We received an invitation and are eager to show you the results of our first tests soon. But be aware that at the moment, you may only try out the Firefly text-to-image generator and play around with creating different text effects. Adobe plans on introducing all the other AI-based features described in this article later this year. We’ll keep you updated!

Conclusion

The more announcements we hear, the more it looks like we’re entering a completely new era in video production. With all these artificial intelligence tools (not only generative AI but also some others that are already taking over mundane tasks), the pre-and post-production processes may speed up much more than we ever thought possible. At the same time, it opens up completely different questions, which have yet to be answered.

And how do you feel about it? Can you imagine implementing the new Adobe Firefly features into your usual workflow? Which one are you looking forward to the most? Let us know in the comments below!

Feature image credit: Adobe.